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ABSTRACT 

Unmanned  vehicles  or  robots  promise  to  increase  the  operational  range  of  the  armed  forces  while 
reducing  the  exposition  of  personnel  to  hazardous  conditions.  They  have  traditionally  been  in  use  for 
aerial  surveillance  and  ground  based  explosive  ordnance  disposal.  Following  the  ongoing  progress  in 
research  and  technology,  unmanned  vehicles  have  the  potential  to  play  a  much  more  versatile  role  in 
future  conflicts.  The  use  of  such  systems  is  already  expanding  into  reconnaissance,  surveillance  and  target 
acquisition  scenarios.  Virtual  and  Augmented  Reality  technologies  are  utilized  across  a  wide  range  of 
relevant  areas  concerning  the  development,  evaluation  and  operation  of  unmanned  vehicles.  Virtual 
Reality  interfaces  are  of  high  value  for  the  development  process  ranging  from  the  technical  design  phase 
to  the  simulation-based  evaluation  of  the  human-machine  system.  A  particularly  relevant  aspect 
concerning  unmanned  vehicles  represents  the  operational  performance  of  the  human-machine-team.  This 
performance  however,  is  primarily  dependant  on  how  effectively  a  human  operator  can  supervise  and 
control  the  unmanned  system  using  a  given  human-machine  interface.  In  this  paper  we  present  a  task¬ 
centric  approach  to  the  command  and  control  of  unmanned  vehicles  that  differs  significantly  from  the 
traditional  vehicle-centric  teleoperation  method  as  used  in  todays  available  systems.  We  believe  that  this 
approach  can  support  the  acceptance  and  use  of  unmanned  vehicles  by  dismounted  soldiers.  Our  interface 
implementation  features  both  2D  and  virtual  scene  representation  components  to  allow  the  task 
specification  by  the  user. 


1.0  INTRODUCTION 

Unmanned  vehicles  or  mobile  robots  promise  to  increase  the  operational  range  of  the  armed  forces  while 
reducing  the  exposition  of  personnel  to  location-bound  hazardous  conditions  and  risks.  Mobile  robots  have 
been  in  development  for  decades  and  have  improved  their  performance  and  fitness  to  operate  in  indoor 
and  outdoor  environments  ever  since  [9].  While  most  commercially  available  military  unmanned  vehicles 
are  either  aerial  vehicles  (UAV)  used  for  reconnaissance,  surveillance  and  target  acquisition  (RSTA)  or 
ground-based  explosive  ordnance  disposal  robots  (EOD),  mobile  robots  addressing  the  problem  of  RSTA 
in  urban  scenarios  have  become  a  main  interest  of  military  forces  involved  in  urban  conflicts.  In  this  paper 
the  focus  is  on  the  operation  of  unmanned  ground  vehicles  (UGV)  in  short  range  non-line-of-sight 
conditions.  In  this  paper  we  present  a  task  centric  approach  to  the  command  and  control  of  unmanned 
vehicles  that  differs  significantly  from  the  traditional  vehicle  centric  teleoperation  method  as  used  in 
todays  available  systems.  We  believe  that  this  approach  can  support  the  acceptance  and  use  of  unmanned 
vehicles  by  dismounted  soldiers. 

From  the  perspective  of  an  operator  of  an  unmanned  ground  vehicle,  controlling  a  mobile  robot  can  in 
general  be  regarded  as  a  dual-task  problem:  The  first,  being  the  pilot  task  covers  the  aspects  of  the 
unmanned  platform,  e.g.  maintaining  operational  effectiveness,  obstacle  avoidance  etc.  The  second,  being 
the  payload  task  covers  all  aspects  regarding  the  task  objectives  the  unmanned  vehicle  is  used  to 
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accomplish,  e.g.  applying  sensors  for  surveillance  or  searching.  In  today’s  available  robotic  systems  such 
as  for  example  UAVs,  this  leads  to  control  stations  with  at  least  two  operators  being  exclusively  assigned 
for  piloting  the  robotic  platform  and  applying  the  payload.  Figure  1  shows  a  simplified  model  of  a 
computer-based  human-machine-interface  as  used  for  controlling  unmanned  vehicles. 
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Figure  1:  General  Model  of  a  Human-Machine-Interface  for  Teleoperation  or  Supervisory 
Control  of  Unmanned  Vehicles.  The  operator  is  given  the  goals  and  parameters  of  the 
pilot  and  the  payload  task  and  executes  a  standard  control  loop.  The  interface 
receives  the  operator  input  and  translates  it  into  robot  commands  to  address 
the  given  tasks.  The  feedback  information  from  the  robots’  sensors  is  rendered 
by  the  display  of  the  interface  which  is  monitored  by  the  operator. 


The  model  depicted  in  figure  1  outlines  the  scope  of  the  task  of  using  an  unmanned  ground-based  vehicle 
from  the  operators’  perspective.  The  human-machine-interface  and  the  vehicle  represent  the  key 
components  in  this  model.  Based  on  the  capabilities  of  the  unmanned  vehicle  various  tasks  in  different 
operational  environments  can  be  addressed.  However,  in  non-line-of-sight  conditions  the  interface 
represents  the  only  link  between  the  operator  and  the  vehicle.  All  operator  input  must  be  translated  into 
vehicle  commands  by  the  input  components  of  the  interface.  The  display  component  of  the  interface  must 
render  all  feedback  information  necessary  to  provide  the  operator  with  a  sufficient  picture  of  the  robots 
context.  The  operator  must  monitor  all  incoming  information  about  the  state  of  the  vehicle,  the 
environment,  and  the  state  of  the  tasks  to  effectively  supervise  and  control  the  unmanned  system. 

This  model  of  teleoperation  for  unmanned  ground  vehicles  has  some  disadvantages  concerning  the 
integration  into  military  operations  based  on  the  use  of  such  vehicles  by  dismounted  soldiers: 

•  Even  under  the  assumption  that  the  pilot  and  the  payload  task  can  be  interleaved  or  serialized  the 
effective  utilization  of  an  unmanned  ground  vehicle  using  this  interaction  model  will  bind  at  least 
one  operator  full  time.  Further  the  operators’  visual  attention  is  bound  heavily  to  the  interface 
when  supervising  the  operation  of  the  vehicle  platform  or  when  executing  some  payload  specific 
operation,  e.g.  manoeuvring  in  difficult  places  or  the  continuous  adjustment  of  sensors  in  dynamic 
situations. 
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•  Unmanned  ground  vehicles  are  in  general  highly  complex  systems.  Realizing  ergonomic  input 
and  output  components  generates  various  constraints  on  the  human-machine-interface.  Foremost 
the  physical  display  size  is  relevant  if  the  unmanned  vehicle  provides  high  resolution  visual 
information.  A  physically  large  interface  is  in  general  considered  difficult  to  be  used  by 
dismounted  soldiers. 

•  The  performance  of  the  unmanned  ground  vehicle  is  highly  dependant  on  the  operational 
environment.  For  effective  operation,  static  and  dynamic  obstacles  must  be  classified  by  the 
operator  to  identify  what  risk  they  pose  to  the  vehicle.  Dynamic  obstacles  such  as  debris,  foliage 
or  left  over  barbed  wire  to  name  a  few  are  difficult  to  judge  in  their  effect  on  the  mobility  and  can 
easily  immobilize  the  unmanned  vehicle.  Further,  operational  environments  may  vary  from  open 
terrain  to  urban  settings  as  well  as  the  utilized  vehicle  may  vary  in  type  and  payload.  This  leads  to 
the  point  that  the  effective  utilization  of  unmanned  ground  vehicles  requires  well  and 
continuously  trained  specialists  rather  then  on-demand  users. 

•  The  presented  model  of  teleoperation  does  not  scale  well  between  the  number  of  operators  and  the 
number  of  unmanned  vehicles  as  well  as  the  type  of  the  vehicle  (ground  and  air).  Multiple 
operators  or  multiple  unmanned  vehicles  are  of  interest  here.  As  well  as  the  use  of  heterogeneous 
type  vehicles  such  as  unmanned  aerial  and  unmanned  ground  vehicles.  A  classification  of  possible 
operator  and  unmanned  vehicle  teams  can  be  found  in  [8]. 

The  above  named  disadvantages  for  the  model  of  teleoperation  as  outlined  in  figure  1  can  be  partially 
addressed  on  three  levels:  first  the  vehicles’  capabilities  and  robustness  can  be  improved  to  reduce  the 
risks  based  on  the  operational  environment,  further  increasing  the  level  of  vehicle  automation  can  reduce 
the  amount  of  supervision  and  interaction  required  by  the  operator.  [4].  Second,  the  training  of  the 
operator  can  be  improved  and  extended  to  prepare  the  operator  for  a  wider  variety  of  situations.  Third,  the 
human-machine-interface  provides  a  range  of  opportunities  to  generate  either  positive  of  negative  impact 
on  the  overall  human-machine-team  performance. 

In  non-line-of-sight  conditions,  when  the  operator  must  completely  rely  on  the  data  transmitted  from  the 
vehicles’  sensors  the  human-machine-interface  becomes  essentially  the  performance  limiting  factor.  To 
address  the  above  mentioned  disadvantages  of  the  outlined  teleoperation  model,  we  developed  a  user¬ 
centric  human-machine-interface  concept  using  a  declarative  approach  to  teleoperation. 


2.0  CONCEPT  OF  A  DECLARATIVE  INTERFACE  FOR  UNMANNED 
GROUND  VEHICLES 

2.1  Declarative  Control  of  an  Unmanned  Ground  Vehicle 

The  attribute  declarative  and  imperative  have  been  translated  from  the  computer  science  domain  where 
they  are  used  to  classify  programming  paradigms.  Traditional  programming  languages  such  as  C,  Pascal 
and  Ada  are  termed  imperative  programming  languages  as  they  require  the  programmer  to  explicitly 
describe  the  computation  in  terms  of  a  program  state  and  statements  that  change  the  program  state  [1]. 
Declarative  programming  on  the  other  side  describes  “what”  a  result  is  like,  rather  than  “how”  to  create  it. 
This  is  a  different  approach  from  the  traditional  imperative  paradigm  [2]. 

In  our  concept  the  traditional  imperative  paradigm  is  analogous  to  the  model  of  teleoperation  as  shown  in 
figure  1 .  An  operator  imperatively  or  explicitly  commands  an  unmanned  vehicle  to  perform  a  sequence  of 
actions  to  resolve  a  particular  task.  This  method  leads  to  the  disadvantages  as  stated  above.  Translating  the 
declarative  paradigm  to  the  operation  of  unmanned  vehicles  means  that  the  operators’  task  will  be  to 
specify  the  desired  result  the  unmanned  vehicle  should  deliver  rather  than  executing  the  command 
sequence  to  achieve  this  result.  A  simple  example  of  declarative  interaction  is  already  widely  used  when 
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specifying  way-  or  goal  points  and  delegating  the  explicit  command  generation  to  an  automated 
subsystem.  An  important  aspect,  when  considering  declarative  interaction  with  unmanned  vehicles  is  that 
the  explicit  command  generation  for  the  vehicle  itself  is  still  required  as  in  the  teleoperation  model.  This 
will  be  done  by  either  a  specialist  operator  or  by  an  automated  subsystem  translating  the  declarative 
interaction  into  a  command  sequence.  Figure  2  depicts  this  concept  schematically. 
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Figure  2:  Declarative  Human-Machine-Interaction  Model.  The  dismounted  soldier  interacts 
with  the  unmanned  ground  vehicle  using  a  standard  interactive  device  as  for  example  a  PDA. 
The  soldier  does  not  control  the  vehicle  itself  but  declaratively  specifies,  for  example  map- 
based,  the  task  the  vehicle  should  resolve.  The  commands  are  relayed  to  a  mobile  UGV 
command  and  control  station  where  a  specialist  operator  uses  the  standard  model  of 
teleoperation  (see  figure  1)  to  resolve  the  specified  tasks. 


The  central  difference  and  advantage  of  the  declarative  teleoperation  approach  is  that  the  interaction  with 
the  unmanned  vehicle  is  transformed  from  vehicle-centric  to  task-  and  user-centric.  Vehicle  specific 
aspects  of  the  teleoperation  are  no  longer  relevant  for  the  dismounted  soldier  as  the  soldier  can  interact 
with  the  UGV  by  stating  “what”  it  should  do  and  the  specialist  operator  in  the  mobile  UGV  command  and 
control  station  is  responsible  for  the  realization  or  the  response  that  the  given  task  can  not  be  executed. 
This  approach  leads  to  the  possibility  that  the  dismounted  soldier  just  needs  an  interface  to  specify  the 
tasks  to  be  resolve  by  the  unmanned  ground  vehicles.  This  type  of  interaction  with  unmanned  vehicles 
provides  several  advantages: 

•  The  declarative  human-machine-interaction  scales  well  with  physically  small  and  lightweight  user 
interfaces.  Without  the  need  to  display  information  associated  with  the  pilot  task,  since  this  is  only 
relevant  to  the  operator  in  the  command  and  control  station,  the  interface  for  the  dismounted 
soldier  can  be  reduced  to  render  only  task  specific  information.  Starting  with  speech-based 
command  input  to  be  interpreted  by  the  operator  in  the  command  and  control  station,  or  other 
small  graphical  user  interface  devices  like  personal  digital  assistants  (PDA,  as  presently  used  in 
some  Special  Forces  units)  can  already  be  used  effectively. 

•  As  stated  above  the  declarative  approach  to  the  use  of  unmanned  vehicles  by  dismounted  soldiers 
can  be  realized  on  small  and  lightweight  interfaces.  Declaring  tasks  for  mobile  systems  essentially 
requires  georeferenciation  as  well  as  action  description.  A  task  description  could  for  example  be 
stated  like  the  following:  “Observe  the  eastern  side  of  building  X  and  report  any  activity”.  This 
task  can  be  well  described  by  text  or  if  available  by  a  graphical  map-based  input  method.  The 
required  input  and  output  modalities  for  this  type  of  interaction  are  already  available  in  today’s 
PDAs,  tablet  PCs  or  other  portable  computers  and  should  therefore  be  well  integrateable  into 
existing  computing  equipment. 
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•  As  the  declarative  approach  of  interaction  with  unmanned  vehicles  is  user-  and  task-centric 
instead  of  vehicle-focused,  it  scales  well  with  multi-operator  and  multi- vehicle  scenarios. 
Developing  effective  means  and  interfaces  for  the  simultaneous  teleoperation  of  multiple 
unmanned  vehicles  is  a  difficult  problem  and  presently  a  research  activity  [5][6].  With  this 
approach  a  dismounted  soldier  is  capable  of  utilizing  multiple  vehicles  simultaneously  by 
specifying  a  task  that  requires  the  use  of  multiple  vehicles.  For  example  the  specification  of  a  task 
to  observe  a  building  from  multiple  angles  to  identify  possible  risks.  This  task  would  be  resolved 
by  multiple  vehicles  each  under  the  supervision  of  a  single  remote  operator.  Concerning  the  multi¬ 
operator  capability,  a  declarative  task  specification  can  be  well  visualized  by  standard  graphical 
user  interfaces.  This  supports  the  co-operative  work  of  multiple  operators  as  they  can  collectively 
work  on  a  given  task  description  as  well  as  identifying  the  motivation  of  a  visualized  task. 

•  The  declarative  interaction  approach  results  in  an  abstraction  from  a  specific  unmanned  vehicles 
by  moving  the  vehicle  specific  command  and  control  aspect  from  the  dismounted  soldier  to  the 
specialist  operator  in  the  UGV  C2  station.  This  allows  the  abstraction  from  a  specific  unmanned 
vehicle  type.  Whether  the  vehicle  is  wheel-  or  track-driven,  or  even  a  legged  device  has  no  impact 
on  the  way  the  dismounted  soldier  interacts  with  it.  The  soldier  just  needs  to  describe  the  “what  is 
to  be  done”  and  the  “how”  is  resolved  by  the  remote  specialist  operator.  A  dismounted  soldier 
without  any  training  in  controlling  unmanned  aerial  vehicles  could  for  example  specify  a  task 
where  the  roof  of  a  specific  building  is  to  be  inspected  for  any  hostile  activity. 

•  We  believe  that  the  declarative  interaction  method  should  require  a  lower  learning  and  training 
effort  than  the  standard  teleoperation  method  as  depicted  in  figure  1 .  Our  believe  is  based  on  the 
following  assumptions:  First  the  declarative  approach  can  be  realized  using  standard  user 
interfaces,  which  are  in  most  cases  well  known  to  many  users.  Second  the  inherent  abstraction  of 
the  specifics  of  the  unmanned  vehicle  of  this  approach  relieves  the  dismounted  soldier  to  aquire 
any  in-depth  knowledge  about  various  unmanned  vehicles.  Third,  the  delcarative  task 
specification  methods  can  be  grounded  on  common  military  types  of  georeferenciation  and  action 
description.  This  should  relieve  the  dismounted  soldier  of  the  need  to  learn  special  and  most 
probably  vehicle-specific  protocols  to  effectively  utilize  an  unmanned  vehicle.  However,  this  is 
only  valid  for  the  dismounted  soldier.  The  UGV  specialist  operator  in  the  command  and  control 
station  responsible  for  executing  the  tasks  as  described  by  the  dismounted  soldiers  must  of  course 
have  in-depth  knowledge  about  the  controlled  unmanned  vehicle. 

•  Scaling  with  increasing  levels  of  vehicle  automation.  Given  the  present  pace  and  progress  in 
research  and  development  of  unmanned  vehicles  one  can  assume  that  the  degree  of  automation,  or 
autonomy  [4]  will  increase  to  either  reduce  the  operators’  workload  and  increase  the  capabilities 
of  the  vehicle.  From  a  dismounted  soldier’s  point  of  view  using  a  declarative  approach  to  human- 
machine-interaction  as  presented  in  this  paper,  improvements  in  automation  based  capabilities  do 
not  change  the  way  of  interaction.  Changes  induced  by  modified  capabilities  of  the  vehicle  affect 
the  specialist  operator  but  are  transparent  to  the  declarative  method  of  interaction  as  used  by  the 
dismounted  soldier. 

2.2  Interface  Realization 

Traditionally,  human-machine-interfaces  for  the  command  and  control  of  unmanned  vehicles  have  been 
vehicle-centric  [3].  In  these  interfaces  a  lot  of  emphasis  is  put  on  the  visualization  of  the  status  of  the 
vehicle  platform  and  the  payload  components  such  as  sensors  and  manipulators.  For  the  dismounted 
soldier  using  an  unmanned  vehicle  in  the  sense  of  using  a  resource  most  if  not  all  of  this  status  information 
is  irrelevant. 

Following  the  declarative  approach  to  the  command  and  control  of  unmanned  vehicles  the  design  of  the 
interface  can  be  task  oriented.  The  task  for  the  dismounted  soldier  as  the  user  of  an  unmanned  vehicle  is 
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no  longer  the  manual  control  of  the  vehicle  but  the  specification  of  what  task  the  vehicle  should 
accomplish  in  the  present  situation.  Designing  an  interface  for  declarative  human-machine-interaction  is 
therefore  more  focused  on  task  specific  input  and  output  methods.  In  our  first  version  of  the  declarative 
interface  we  chose  therefore  a  standard  tablet  PC  as  input  and  output  device  for  the  dismounted  soldier. 

The  input  modality  used  in  our  example  is  pen-based  interaction  as  this  is  a  likely  form  to  be  used  when 
requirements  for  physical  dimension,  mobility  and  speed  of  input  do  not  permit  keyboard  input  methods. 
The  output  modality  is  primarily  visual  with  supplemental  auditory  elements.  An  interface  for  declarative 
interaction  requires  primarily  the  referenciation  of  both  locations  and  objects  to  define  tasks  and  task 
constraints  such  as  directions,  distances  and  timing.  To  enable  the  declarative  task  specification  we  used 
an  interface  based  on  the  sketchboard  metaphor.  Sketchboards  are  already  widely  used  as  rapid 
prototyping  interfaces.  We  use  the  sketchboard  to  allow  the  use  the  specification  of  tasks  by  drawing  the 
task  parameters  into  a  map  from  a  top  down  two  dimensional  perspective.  A  secondary  input  and  output 
component  is  realized  by  the  visualization  of  a  virtual  environment  of  the  task  scene  from  either  a  vehicle 
based  perspective  or  an  arbitrary  virtual  location  to  allow  the  user  further  means  of  specifying  task 
constraints.  Figure  3  shows  a  sample  screenshot  of  the  interface  as  seen  by  the  user. 


Figure  3:  Screenshot  of  the  Tablet  PC  Based  Interface.  The  lower  section  contains  the 
sketchboard  interface  showing  three  unmanned  ground  vehicles  as  arrow-head  shaped 
elements.  Three  simplified  buildings  are  present.  The  sketchboard  contains  some  icon-based 
mode  buttons  to  enable  standard  create/edit/delete  operations  by  the  user.  The  top  section  of 
the  interface  contains  the  rendering  of  the  virtual  environment  which  is  generated  out  of  the 
map  data  as  present  on  the  sketchboard.  The  perspective  in  this  example  is  based  on  the 
camera  of  the  rightmost  vehicle  showing  the  other  two  vehicles  in  front. 


2.3  Declarative  Task  Specification 

An  important  benefit  of  the  declarative  interaction  with  unmanned  vehicles  is  that  it  can  be  realized  using 
standard  computing  equipment  using  well-known  interactive  methods  such  as  the  pen-based  sketchboard 
approach  we  use  in  our  example.  The  same  is  valid  for  the  process  of  the  declarative  task  specification 
which  has  to  be  done  by  the  dismounted  soldier.  Standard  procedures  known  from  graphical  user 
interfaces  such  as  point  and  click,  component  dragging,  undo  and  redo,  or  opening  a  context  menu  can  be 
used  for  the  specification  of  a  task  for  the  unmanned  vehicle.  Figure  4  depicts  a  sample  task  specification 
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following  the  situation  as  present  in  figure  3.  The  interaction  in  this  example  is  conducted  using  only  pen- 
based  input. 


Figure  4:  Sample  Task  Specification  Using  the  Sketchboard  Interface.  In  this  example  the  user 
(dismounted  soldier)  wants  to  observe  the  leftmost  building.  To  move  two  vehicles  into  the 
desired  observation  position  a  path  is  drawn  on  the  sketchboard  to  be  following  by  the  vehicles. 
The  relevant  building  is  marked  on  the  map  and  using  the  context  menus  for  building-based 
operations  the  user  specifies  the  task  to  be  resolved.  In  this  example  the  user  can  choose 
between  “Observe”,  “Examine”,  “Investigate”. 


3.0  CONCLUSION 

In  this  paper  we  stated  that  the  standard  model  of  teleoperation  as  shown  in  figure  1  has  several  drawbacks 
when  concerning  the  use  of  unmanned  vehicles  by  dismounted  soldiers.  These  drawbacks  are  in  general 
founded  in  the  requirement  that  operating  an  unmanned  vehicle  is  a  high  workload  task  which  binds  a 
single  person  full  time  to  a  complex  human-machine-interface.  To  improve  the  possibility  of  the 
integration  and  use  of  unmanned  vehicles  with  teams  of  dismounted  soldiers  we  developed  the  concept  of 
a  declarative  human-machine-interface.  This  concept  differs  from  the  traditional  vehicle  centric  paradigm 
of  the  command  and  control  of  unmanned  vehicles  as  it  focuses  on  task  descriptions.  We  believe  that  this 
approach  has  several  advantages  compared  to  the  traditional  approach: 

•  It  allows  for  simple  and  lightweight  user  interfaces  based  on  standard  hard-  and  software. 

•  It  scales  well  with  multi-operator  and  multi-vehicle  scenarios  as  the  task  declaration  is  based  on 
the  desired  result  and  therefore  invariant  to  the  number  of  vehicles  used  to  achieve  this  result. 
Even  mixed  type  vehicles  such  as  UGVs  and  UAVs  can  be  used  this  way. 

•  Possibly  lower  training  requirements  since  the  interface  can  be  built  around  standard  input  and 
output  components  as  used  in  todays  portable  computing  equipment.  Also  the  vehicle  abstraction 
should  reduce  the  required  amount  of  specific  knowledge  about  particular  models  of  unmanned 
ground  vehicles. 

•  It  scales  well  with  different  levels  of  vehicle  based  automation.  Due  to  the  vehicle  abstraction  of 
the  interface  different  levels  of  automation  should  be  transparent  to  the  user. 
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These  advantages  should  support  the  acceptance  and  use  of  unmanned  vehicles  by  dismounted  soldiers  in 
missions  where  such  vehicles  can  take  over  hazardous  activities.  The  next  step  in  our  development  process 
is  the  actual  testing  and  evaluation  of  our  interface  with  novice  subjects.  The  results  of  this  evaluation  will 
guide  the  next  development  cycle.  At  this  stage  the  three  dimensional  scene  representation  is  extended  to 
enable  the  creation  and  modification  of  tasks  from  multiple  perspectives. 
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